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In the United States, federal and state accountability policies put demands on educa¬ 
tional agencies that often exceed their capacity. In 2001, No Child Left Behind legis¬ 
lation required 100 percent of students to achieve proficiency by 2014. As the 
deadline rapidly approached, the federal government has offered waivers for exemp¬ 
tion from policy requirements, but with demanding expectations for improvement 
(USDOE, 2012). Concurrently, Race to the Top funds were made available through a 
grant competition, offering states substantial funds during difficult economic times 
to support improvement efforts, but requiring that “ambitious” strategies and policies 
be adopted in order to qualify. In this context, schools, districts, and states nationwide 
are under increased pressure to make dramatic changes to ensure achievement of pro¬ 
ficiency by all students. These demands often exceed the capacity of education agen¬ 
cies to design and implement ambitious reform, particularly as state education 
agencies have experienced declining budgets and staff (CEP, 2011; Jochim & Murphy, 
2013). One response to this overextension is for education agencies to turn to a range 
of external organizations (Datnow & Honig, 2008; Honig, 2004; Jacobson, 2008) for 
support—what Rowan (2002) referred to as the “school improvement industry”— 
which includes for-profit vendors, technical assistance agencies, universities, nonprof¬ 
its, and other types of organizations. These organizations are given a contract by ed¬ 
ucational agencies to design, implement, or evaluate improvement initiatives. 

Contracted organizations are referred to by a variety of names—intermediary 
organizations, vendors, technical assistance providers—and are described by the 
purposes they serve (see Honig, 2004, for a more thorough discussion). Research 
argues that such organizations can play a powerful role in mediating policy (Coburn, 
2005; Honig, 2004), yet most studies to date have typically focused on these organ¬ 
izations as “background” in implementation research (Honig, 2004). Overall, the re¬ 
search on these organizations is sparse (Burch, 2007). Calls for additional research 
emphasize the need to determine if and how interactions between educational agen¬ 
cies and these other types of organizations can influence the design and implemen¬ 
tation of policy. 

The purpose of this article is to explore a secondary role of educational contrac¬ 
tors or vendors. Research on these organizations, though scant, typically focuses on 
policy implementation in the content area for which the contract was given—for ex¬ 
ample, Coburns (2005) analysis of reading instruction in California. However, edu¬ 
cational contractors or vendors are situated to serve a secondary but potentially 
significant role as brokers of research-based knowledge (Davies & Nutley, 2008). 

Theoretical framework and research perspectives 

Research brokers—those who package the outputs of the research community to 
policy makers (Sundquist, 1978)—are important within the context of accountability. 
Policies such as NCLB demand evidence-based decision making and the selection 
of research-based programs in school improvement efforts. Accountability policies 
have elevated the importance of educational research and research-based knowledge, 
yet gaps between research, policy, and practice persist, as educational agencies strug¬ 
gle to dedicate limited resources (time, in particular) and capacity (human, in par¬ 
ticular) to the range of needs that must be met. A number of studies have examined 
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how intermediary organizations can serve to link research, policy, and practice, yet 
these studies typically focus on state or district partnerships with organizations 
whose primary role has been to support the integration of research into design and 
implementation of policy (see volume edited by Coburn & Stein, 2010; Cooper, 
2012; Scott, Lubienski, DeBray, & Jabbar, 2014). The contribution of educational 
contractors to educational policy through research brokering has been only mini¬ 
mally acknowledged (Smylie & Corcoran, 2009; Rowan, 2002; Massell & Goertz, 
2012) or studied in the field of education, yet the potential for these organizations 
to both extend the capacity of educational agencies to implement reform and to 
translate research into policy and practice demands further attention. 

This article contends that educational contractors can influence the use of re¬ 
search in a) the design of policy and b) in the implementation of policy. When seek¬ 
ing supporting services, educational agencies can offer funds to educational 
contractors, typically on a competitive basis, through a request for proposals (RFP) 
process. Proposal responses to the RFPs detail programs or services that may or may 
not be evidence-based or draw on educational research. The successful respondent 
therefore shapes the research base (or lack thereof) behind the final policy imple¬ 
mented. Then, upon receiving the contract, vendors are responsible for implement¬ 
ing their evidence-based plan with fidelity, supporting the translation of 
research-based policy into practice. In cases where RFPs are loosely constructed, 
with content and delivery determined by the contractors, educational contractors 
are well positioned to influence policy and practice through the evidence base of 
their programs or services. In cases where RFPs are tightly constructed, with content 
and delivery predetermined by the educational agency, then educational contractors 
are relatively less influential in serving as research brokers in educational policy. 

Given the role of educational contractors in translating research into both policy 
and practice, there exists a need to consider the degree and nature of research use 
by these organizations. We direct attention to this oversight by way of exploring the 
case of the data coach initiative in Delaware, guided by the following questions: 

1. Are programs/services offered by vendors based in research evidence? 

2. What type of research is influential in vendors’ services? Specifically, what 
sources, methods, and other characteristics of research are utilized? 

3. Is there a core body of research that influences vendor programs/services? 

4. Is there any relationship between vendor/proposal characteristics 
and research use? 

Context of study 

Delaware was among the first awarded a Race to the Top grant, in which the 
Delaware Department of Education (DEDOE, 2010) proposed an initiative com¬ 
monly referred to as “data coaches.” The RFP was loosely constructed in the sense 
that there were few prescribed elements to the planned initiative: 

1. Following a proven approach to using data to inform instruction 

2. Analyzing, drawing conclusions from data 

3. Facilitating collaborative planning sessions to build technical and 
pedagogical skills and providing feedback 
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4. Being flexible and responsive to build strong relationships and a cul¬ 
ture that values open discussion and responsiveness to data 

This left substantial room for vendors to develop new programs or to propose 
previously developed programs. Furthermore, the RFP included a section entitled 
“Evidence of Effectiveness,” specifically requesting “the research base for your 
methodology.” This section accounted for 30 percent of the total score assigned each 
vendor, constituting the criteria with the greatest weight in the evaluation process. 1 

The issue of data coaches is also a salient one, as most education agencies have 
emphasized increased data use in order to improve instruction, and assessment/data 
represents a growth industry in education (Burch, 2006). Additionally, research related 
to data use has a number of characteristics, which make it ideal for an exploration of 
vendors as research brokers. First, the body of research specific to data use is relatively 
small, in comparison to, for example, reading instruction. Thus, the pool of research 
is bounded and its uptake can be assessed more easily than in other areas. Second, 
data use has been connected to a number of other potentially relevant literatures, in¬ 
cluding leadership, school culture, professional development, and effective instruc¬ 
tional strategies. These secondary bodies of literature are added to the pool of research, 
offering an opportunity to examine whether educational contractors are drawing on 
a broader or narrower set of research when developing programs and services. 

Methods and data 

We use a bibliometric strategy with a combination of content and citation analysis 
to investigate the characteristics and quantity of research evidence (broadly con¬ 
strued) utilized by vendor responses to the data coaches RFP. Data are drawn from 
the population of vendor responses (N = 14) to the state-issued RFP Though there 
is no way to ascertain the representativeness of this pool of applicants, we believe 
that this strategy is both a reasonable approximation in the absence of a clear sam¬ 
pling frame, as well as appropriate for the exploratory nature of the study. 

Research, as defined by the Cambridge Dictionary, is “a detailed study of a sub¬ 
ject in order to discover information or achieve a new understanding of it.” To this 
end, we consider all materials referenced in support of the knowledge base underly¬ 
ing vendor services to be potential sources of research evidence. Though narrower 
definitions might be applied, our purpose is to explore the nature of the evidence 
base supporting educational vendors; bounding the definition of research would 
prohibit a full description of that underlying body of knowledge. 

Data 

Cited research measures. Although the RFP indicated several sections of the proposal 
in which citations would be appropriate, we initially sought to document research 
use under the section entitled “evidence of effectiveness.” Flowever, only 96 (49%) 
of all references were used in this section of the proposal. This excludes more than 
half of references and may not actually reflect vendors’ use of research evidence. For 
example, in one proposal under the “evidence of effectiveness” section, the vendor 
directed the reader to earlier parts of the proposal in which a rich body of research 
was presented. In this case, only one of many references was actually cited as evi- 
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dence, when in fact all references served in this capacity. Because of this being a po¬ 
tential issue across all proposals, made explicit or not, we refocused analyses on all 
references, interpreting findings as liberal estimates of the use of research. Reference 
lists, footnotes, and undocumented references to research were extracted from each 
proposal to create a dataset of research evidence employed by the vendors. 

To categorize the type of research evidence utilized, either the abstract or com¬ 
plete reference were located and examined by one member of the research team, 
with member checks on agreement conducted by the lead researcher to ensure reli¬ 
ability. We initially categorized references as having a primary focus that is empirical, 
defined as based on systematic observations or data; review of literature, defined as 
synthesizing existing theory or work; or conceptual, theoretical, or advocacy, defined 
as presenting a non-empirical perspective on an educational issue. This latter cate¬ 
gory also included prescriptive literature. Empirical references were then coded in 
an a priori framework organized around non-mutually exclusive categories of quan¬ 
titative, qualitative, survey, evaluation of an intervention, and case study. An addi¬ 
tional category of empirical evidence emerged, which we label “data”; these included 
references that supplied data without significant analysis or interpretation (e.g., state 
data websites). All references were categorized by publication venue or source (jour¬ 
nal, book, chapter in book, published report, conference paper, dissertation, or 
other), and their use in the proposal (located under “evidence of effectiveness” or 
other section). We additionally coded each reference by primary research topic, as 
the issue of data use is often tied to multiple fields of inquiry. These codes developed 
through an emergent coding process in which one member of the research team gen¬ 
erated a list of key topics covered by each reference. The second member of our 
team conducted a member check and developed a collapsed set of categories cover¬ 
ing a broader range of research topics. The identified topics were: 

Teacher or principal education. References focused on issues related 
to teacher or principal preparation programs. 

Achievement/achievement gap. References focused on analysis of 
student achievement, racial or other achievement gaps, or student 
performance at school, district, state or national levels. 

Teacher quality, effectiveness, or evaluation. References focused on 
the measurement or predictors of teacher quality or effectiveness, 
or references focused on performance evaluation of teachers. 

Professional development/coaching. References focused on evidence 
of teacher learning, methods and topics of professional develop¬ 
ment, and coaching models for improving instructional capacity. 
Instructional strategies/pedagogical content knowledge. References fo¬ 
cused on teacher knowledge or practices, but not the develop¬ 
ment of that knowledge/practice. 

Leaming/instructional theory. References describing, summarizing, 
or synthesizing theories of teaching and learning. 

Other theory/frameworks. References describing applied frame¬ 
works or implementation of theory; for example, research dis¬ 
cussing Response to Intervention frameworks. 
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School and community contexts. References focused on systems, 
structures, and cultures within schools, including teaming, profes¬ 
sional learning communities, school-community partnerships, 
and distributed leadership. 

School improvement, reform, and policy. References focused on 
school improvement strategies or programs, educational reform, 
and educational policy in general. 

Data use. References focused on educators’ use of data or pro¬ 
grams to foster educators’ use of data. 

Assessment. References focused on issues of measurement, types of 
assessments, or assessment frameworks. 

Many references draw conclusions about or include implications for many of these 
topics; however, we applied categories based only on the primary purpose of the research. 

Vendor characteristics. A database of vendors was also created, with variables 
describing the organization type/affiliation, organization size, proposal budget, and 
educational background of key personnel (see Table 1). These characteristics were 
directly attainable from the RFP responses, as they were required components of the 
proposals. Organization size and budget/cost are continuous variables; however, be¬ 
cause of the small sample (N = 14) and wide range of values, we transformed these 
into categorical variables based on quartiles to generate groups of cases. The other 
variables were categorical in nature. Organization type was categorized as for profit, 
nonprofit, and university based. Educational background was determined by 
whether any of the project leadership identified in the proposal had either an EdD 
or PhD. This was unavailable only in one case. 
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Table 1. Descriptive statistics of vendor measures 



N 

% 

Organization type 

for profit 

12 

85.7 

nonprofit 

1 

7.1 

university based 

1 

7.1 

Education of leadership 

With PhD 

6 

42.9 

With EdD 

3 

21.4 

With Master’s or lower 

4 

28.6 

Not available 

1 

7.1 


Mean 

SD 

Proposed budget cost 

6,610409 

7,676,314 

Quartile 1 (n = 5) 

1,225,348 

766,976 

Quartile 2 (n = 2) 

1,655,429 

1,499,673 

Quartile 3 (n = 4) 

7,317,155 

1,110,344 

Quartile 4 (n = 3) 

17,946,502 

9,383,968 

Organizational size (employees) 

751 

1220 

Quartile 1 (n = 4) 

20 

16 

Quartile 2 (n = 3) 

79 

39 

Quartile 3 (n = 4) 

450 

282 

Quartile 4 (n = 3) 

2800 

1153 
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Analysis 

Research characteristics data were initially analyzed descriptively using frequencies 
and cross-tabulations to answer our first two research questions, which focus on the 
characteristics of research utilized by vendors in this study. Additionally we utilized 
a social network analysis approach to examine our third research question, which 
focuses on the core body of research influencing services offered (see Borgatti, Mehra, 
Brass, & Labianca, 2009, for a discussion of network analysis in social sciences). A 
social network approach examines relationships between actors in a network and 
has been applied to bibliometric analyses to examine the impact of and relationship 
between particular research articles, scholars, and journals through citation networks. 
We adopt a particular strategy similar to co-citation analysis (White & Griffith, 1981), 
which can be used to identify authors citing similar references or to identify connec¬ 
tions between research disciplines. Applied to the study of research use by educa¬ 
tional contractors, these methods can determine the relative importance of each 
reference as well as the relationship between vendor proposals based on common 
references. We conceptualize the field of vendors as a sample network of professional 
development providers that provide services related to schools’ and teachers’ use of 
data in educational decision making. Concurrently, we consider the range of research 
employed in vendor proposals as the population of research influencing practice. 
Bringing these two sets of data together, we generated a two-mode matrix connecting 
research to vendors. We utilized UCINet and Netdraw (Borgatti, Everett, & Freeman, 
2002) to analyze and transform this network. 

We first produced a two-mode matrix in which research references and vendors 
are nodes. In this network, vendors are tied to their references, which may be tied 
to other vendors if both cite the mutual reference. Within this network, we are in¬ 
terested both in the characteristics of the network as a whole and in the network po¬ 
sition of both vendors and research. We utilize measures of network density and 
fragmentation to assess the nature of research utilization by vendors in this field. In 
such a network, dense ties—as measured by the proportion of ties possible that ac¬ 
tually occur—would indicate that the population of vendors is utilizing a common 
body of research and, by extension, that services would be similarly based in that 
body of work. Fragmentation—as measured by the proportion of the network that 
cannot “reach” each other through ties—would indicate that there are disconnected 
vendors or groups of vendors who do not share any references in common. For ex¬ 
ample, high fragmentation would indicate that the population of vendors draws on 
distinct research bases when providing services. 

In addition to the characteristics of the network as a whole, we also are interested 
in the network position of vendors and research references. That is, within the struc¬ 
ture of the network, where is each vendor and reference located? To identify a “core” 
body of research that influences data use programs and services, we focus on degree- 
based centrality, which measures how important a reference is by the number of ties 
it has to vendors. A more central reference would be considered more influential 
than less central ones. Each reference’s “degree” centrality is then examined in con¬ 
junction with other reference characteristics (e.g., topic, method) to determine the 
characteristics of the most influential research. 


Farley-Ripple 
& Jones 

Research into 
Practice 


IJEPL 10 ( 2 ) 2015 


We similarly consider the degree centrality of vendors. To do this, we performed 
an affiliations transformation (multiplying the matrix by its inverse) to create a net¬ 
work in which each vendor is a node and the ties between nodes indicate their uti¬ 
lization of a common reference. That is, a tie indicates that two vendors had a 
reference in common. Within the vendor network, we calculated the degree centrality 
of each vendor and added this to the set of characteristics of vendors used to answer 
the fourth research question. To assess the association between vendor characteristics 
and the use of research, we conduct correlation and compare means across vendor 
types and characteristics. 

Limitations 

There are a number of limitations to this research design. First, the sample is limited 
to the data coach initiative, which may not be representative of other competitions 
and may relate to bodies of research that have significantly different characteristics 
than other research areas. Relatedly, the population of vendors is small, which may 
not be sufficient to understand the distribution of characteristics across the larger 
population of vendors or references, and which results in insufficient power to detect 
statistically significant differences. Third, we operate on the assumption that research 
cited in proposals represents good faith efforts to draw on existing evidence in pro¬ 
gram design and delivery, rather than strategic inclusion of well-known works or 
large numbers of references to symbolically comply with RFP requirements. Lastly, 
the set of vendor and research characteristics included in analysis may not represent 
all potentially important attributes associated with the brokering role of educational 
vendors. We therefore offer results and conclusions with the caveat that additional 
research is needed to explore the significance of these limitations. 

Results 

Research Base of Programs and Services 

Of the 14 proposals, 12 (85.7%) provided citations indicating their services were 
supported by research evidence, and two (14.3%) proposals had no citations indi¬ 
cating a research base at all. A total of 197 references were cited by the collective 
group of vendors, though vendors varied widely in the degree to which they refer¬ 
enced research, ranging from two research resources to 49. The mean number of ref¬ 
erences was 17.5, with a standard deviation of 15.04 and a median of 12.5 
(excluding those who had no references cited). 

As explained earlier, the types of research referenced in vendor responses to the 
RFP were analyzed in terms of whether they were considered empirical, and if so, they 
were then analyzed in terms of the methodology employed, general research focus, 
and publication source. The results of these analyses are presented in Figures 1-4. 

Results in Figure 1 illustrate that empirical research is most frequently cited in 
support of vendor services, though it does not constitute the majority of references, 
nor is it the type of reference vendors used most. Literature reviews constitute the 
smallest proportion of all references, but are cited by as many vendors as empirical 
research. Further, though theoretical, conceptual, and advocacy work does not con¬ 
stitute the largest proportion of references, this category of research is referenced by 
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the majority of vendors. These findings offer evidence that vendors do not utilize re¬ 
search in similar ways. Rather, it seems that fewer numbers of vendors make use of 
many empirical studies, while more vendors rely on fewer literature reviews and the¬ 
oretical works. 
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Figure 1. Distribution of references by nature of research 
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The distribution of empirical references by method (Figure 2) reveals relatively 
equitable distribution of studies with varying characteristics, ranging from 13 percent 
for evaluations to 21 percent for qualitative studies. The proportion of references sug¬ 
gests that there is no particularly dominant methodology utilized by vendors as a 
whole. Similarly, the proportion of vendors referring to each type of research is more 
balanced than analyses based on other characteristics, suggesting that vendors do not 
vary substantially in their preferred type of research evidence. However, data sources, 
such as state data websites, were less often used and were used by only one vendor. 


Figure 2. Distribution of empirical references by method 
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In Figure 3, results indicate that research available as published reports com¬ 
prises the largest portion of references used among vendors, with a nearly 20 percent 
difference from the next most frequent publication source (journals). Books and jour¬ 
nals each constitute approximately a fifth of the references, with chapters, conference 
papers, internal reports, and dissertations all constituting less than 10 percent. 
“Other” publication sources included references to websites, webinars, and DVDs, 
but were few in number. In comparison to the proportion of references constituted 
by each source, the proportion of vendors citing each source is quite different. More 
than three quarters cite books, and a majority cite published reports, journal articles, 
and other sources. This suggests that, while some sources are fewer in number, they 
enjoy great popularity, as evidenced in how many vendors utilize them. Thus, the 
quantity of each type of reference is not an indicator of how central they might be 
in vendors’ use of research. 


Figure 3. Distribution of references by publication source 
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Figure 4 presents the distribution of references by research topic. Flere we see a 
wide range of topics presented, with no clear field of study emerging as most influential. 
Among the most frequently cited are professional development/coaching; instructional 
strategies/PC; data use; and school improvement, reform, and policy, all constituting 
more than 10 percent of the set of references. These are also more likely to be cited by 
a majority of vendors. Less frequent and influential topics include teacher/principal 
education, teacher quality, leaming/instructional theory, and other theories/frameworks. 
Interestingly, there are a few topics that constitute a small portion of the references but 
are popular among vendors, including assessment, the achievement gap, and school 
contexts. This indicates that a broad range of research may be influential across many 
vendors, regardless of whether there are many or few references on the topic. In fact, 
proportionally, those references that constitute the smallest proportion of all references 
but are cited by a majority of vendors may be the most influential, in contrast to a 
larger body of work cited across multiple vendors. 
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Figure 4. Distribution of references by topic 
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Network characteristics 

We build on these descriptive findings through network analyses. Figure 5 illustrates 
the two-mode network of vendors and research. Vendors are represented by white 
circles, and research references are represented by grey squares. Figure 5 provides a 
visual illustration of many aspects of the research-vendor network captured in our 
data. As discussed in our methods section, we focus on network characteristics of 
density and fragmentation, as well as centrality for research references. 


Figure 5. Two-mode network of research and vendors 



The two-mode network has a density of .088, which means that about 9 percent 
of the ties possible within the network are actually present. In the context of this 
analysis, this means that there are few ties—that is, references—common between 
vendors. The low density of this network is visually evident in Figure 5 and indicates 
that there is not a commonly identified body of research on which vendor services 
are based. Not surprisingly, fragmentation in this network is substantial, at .402, 
which means that 40 percent of the network is not reachable through other ties. In 
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the context of this analysis, this means that a significant portion of research references 
and vendors have no connection to other research and vendors. This is visually pre¬ 
sented above, as two vendors and their corresponding references at right have no 
connection—common references—with other vendors. On the left side of Figure 5, 
we see the remaining ten vendors, who are connected by at least one tie or common 
reference. Notably, however, the connections between vendors are tenuous, with 
very few common research resources cited. Only one pair of vendors shares more 
than one reference in common, with the remainder tied to other vendors via a single 
common reference. 

In fact, only eight references (labeled in Figure 5) were mentioned by multiple 
vendors: four were referenced by three vendors (centrality = .25), and four were ref¬ 
erenced by only two vendors (centrality = .17). Those referenced by more than one 
vendor will have greater degree of centrality, which we interpret as a measure of in¬ 
fluence on the data use services field. Central references were more likely to focus 
on data use (6 of 8), to be non-empirical (5 of 8 were conceptual/prescriptive, 1 of 
8 was a literature review), and to be books (4 of 8), in contrast to the proportion of 
these characteristics in the larger population of references. 

Those that emerged as central are not surprising for a few reasons. First, an em¬ 
phasis on data use might be expected to be common across vendors, given the pur¬ 
pose of the RFP. Second, several of these common references reflect popular tools in 
practice. For example, Data Wise is a popular prescriptive program for using data 
to improve schools (Boudett, City, & Murnane, 2005; Boudett & Steele, 2007), so 
this reference may be more likely to be recognized by professional developers creat¬ 
ing programs as well as recognized by the state and practitioners adopting vendor 
services. Similarly, Danielson’s (2007) framework has been adopted by many local 
education agencies to develop and evaluate teachers. Third, the centrality of non- 
empirical work is telling. This may confirm recent claims that “the practice of data 
use is out ahead of research. Policy and interventions to promote data use far outstrip 
research studying the process, context, and consequences of these efforts” (Coburn 
& Turner, 2011, p. 200). At the time the RFP was issued, substantial research was 
underway investigating educators’ use of data, but in general the field is in its infancy, 
working toward a clear body of knowledge to inform practice. However, the avail¬ 
ability of empirical studies, not only of data use, but of other related topics (e.g., 
coaching, professional development, teacher collaboration), appears underrepre¬ 
sented among central references, and this is indicative of weak links between re¬ 
search, policy, and practice. 

Relationship between research use and vendor characteristics 

Our final research question focuses on whether there are any relationships between 
the characteristics of vendors and the characteristics of research references used. 
Because 12 of the 14 vendors are for profit, we focused our analyses on characteris¬ 
tics of organization size, budget cost, and education level of project leadership in re¬ 
lation to six measures of research use: number of references; proportion that are 
empirical, literature review, or conceptual/advocacy/theoretical; proportion used as 
evidence of effectiveness; and degree centrality. Table 2 presents a comparison of the 
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Table 2. Relationship between research use and vendor leadership education 2 



Mean 

SD 

Number of references 

No doctorate 

10.40 

11.84 

PhD 

18.67 

16.21 

EdD 

15.67 

22.14 

Total 

15.07 

15.30 

Proportion of references: 
empirical 

No doctorate 

0.26 

0.24 

PhD 

0.51 

0.18 

EdD 

0.24 

0.34 

Total 

0.38 

0.24 

Proportion of references: 
literature review 

No doctorate 

0.21 

0.11 

PhD 

0.18 

0.18 

EdD 

0.07 

0.10 

Total 

0.17 

0.14 

Proportion of references: 
theoretical, conceptual, or advocacy 

No doctorate 

0.51 

0.19 

PhD 

0.40 

0.19 

EdD 

0.68 

0.45 

Total 

0.48 

0.24 

Proportion of references used 
as evidence of effectiveness 

No doctorate 

0.69 

0.36 

PhD 

0.33 

0.28 

EdD 

0.96 

0.05 

Total 

0.55 

0.37 

Degree centrality 

No doctorate 

2.00 

1.22 

PhD 

2.67 

1.75 

EdD 

2.00 

3.46 

Total 

2.29 

1.90 


number of references, proportion of references of different types, and proportion of 
references used as evidence of effectiveness by education level of vendor project lead¬ 
ership. Only one relationship was statistically significant, with vendors led by staff 
with no doctorate, or with an EdD, more likely to use references as evidence of ef¬ 
fectiveness than those led by staff with PhDs. This pattern is difficult to interpret 
and may be biased due to how evidence of effectiveness was discussed within each 
proposal (see earlier discussion). Table 3 presents Pearson’s correlation coefficients 
examining the same measures of research use by vendor size and budget. There are 
no statistically significant relationships, and only two cells approach a moderate re¬ 
lationship. These relate to organizational size and suggest that larger organizations 
are more likely to use a greater number of references, but less likely to refer to them 
as evidence of effectiveness. 


In general, there appears to be no pattern in the relationships between research 
use and vendor characteristics as measured in this study. However, the sample is 
small and homogenous in terms of whether it is for profit, nonprofit, or university 


13 






























based—a potential indicator of association with the research community as well as 
of resources for conducting internal research and development. Further inquiry that 
tests this particular relationship is needed to understand this potential factor. 


Table 3. Correlation testing relationship between research 
use and vendor characteristics 



Cost 

Cost 

Quartile 

Size 

Size 

Quartile 

Number of references 

-.001 

.243 

.001 

.411 

Proportion of references: empirical 

-.314 

.058 

.207 

.159 

Proportion of references: literature review 

-.028 

-.224 

.203 

.235 

Proportion of references: theoretical, 
conceptual, or advocacy 

.191 

.016 

.028 

-.060 

Proportion of references used as evi¬ 
dence of effectiveness 

.112 

-.143 

-.384 

-.200 

Degree centrality 

.098 

.252 

-.199 

-.060 
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Discussion 

The purpose of this analysis has been to explore the role of educational contractors, 
or vendors, as research brokers who translate research into practice through the de¬ 
velopment and implementation of services in schools. Findings with respect to our 
first research question—which reflects whether services are based on research—sug¬ 
gests that in fact, vendors do appear to base their work in educational research, 
broadly defined. However, the variability among vendors—with two not including 
any evidence base, and some utilizing upwards of 40 different references—suggests 
that not all providers utilize research in their programs, in spite of research evidence 
being a required component of the state’s RFP. This finding brings into question the 
extent to which research is valued by vendors in program development and delivery. 

Further, the variability in the nature of the research base supporting vendor services 
suggests that being “evidence-based” is interpreted broadly. Just under half of refer¬ 
ences were considered to be empirical and nearly a third of vendors did not reference 
a single empirical source. This suggests that programs implemented in schools may 
be only loosely grounded in empirical evidence, if at all. 

In response to our second research question, the research base utilized is notably 
broad in terms of method, topic, and publication source. This finding is highly con¬ 
sistent with the state of data use research, which is primarily descriptive in nature 
and is conceptually linked to many other issues of teaching and learning, such as 
professional community, collaboration, professional development, leadership, and 
school improvement/reform. Additionally, as data use research is an emerging rather 
than highly developed field of study, work is available in a variety of formats, ranging 
from prescriptive literature, to evaluation reports posted to websites, to formal peer- 

reviewed journals. 14 
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Through our third research question, we explore the extent to which there is a 
core body of research that influences vendor programs or services. Although vendors 
utilized a broad range of references in explaining the research base and evidence of 
effectiveness of their services, the core body of research employed—those common 
to multiple vendors—was small and far more homogenous, including primarily non- 
empirical literature and emphasizing data use. There are several potential explanations 
for this finding. First, it may be interpreted as consistent with claims that practice has 
outpaced research for this improvement strategy: the fact that this core set of work 
emphasizes data use may indicate that vendors desired to draw more directly on re¬ 
search related to data use, and that in the absence of a strong body of empirical work 
(at the time of the RFP), they may have relied on popular prescriptive work. In con¬ 
trast, it may also suggest that the relevant literature is broad and complex, such that 
using it for program development could entail exploring only small portions of the 
relevant literature leading to diverse programming in schools. Finally, this finding 
may reflect the effort vendors dedicate to the development of products and to the 
proposal process, signaling search and incorporation of research in less than compre¬ 
hensive and meaningful ways. Irrespective of interpretation, there is little to suggest 
that a common body of research is influencing practice. 

Finally, we considered the possibility that the role of research in vendor services 
might vary by the characteristics of the organization. Evidence presented here does 
not suggest that research use is associated with the size, education level, or cost of 
services. That does not preclude the possibility of other organizational characteristics 
being related to research use—including attributes not observable in this study or 
attributes not captured in this population of vendors. For example, only one vendor 
was affiliated with a university, and one was a nonprofit. In fields with greater num¬ 
bers of vendors of these types, differences may emerge. 

As an exploratory study, the context of this analysis is limited to a single state 
and initiative. Further, the initiative in question—data coaches—is not only linked 
to a broad range of educational research, but is also directly connected to emerging 
work on data use. The nature and quality of related research used by vendors is 
likely to differ by the initiative and by state/district context. We believe further re¬ 
search of this type is needed, and it is our hope that the type of work here motivates 
broader inquiry in other areas influenced by the privatization phenomenon. 

Conclusion 

The current accountability context places demands on educational agencies that often 
exceed their capacity. As a result, a variety of educational organizations are contracted 
to design and implement policy-mandated programs—and such privatization has 
been acknowledged in data use related areas of data management and staff develop¬ 
ment (Burch, 2009). The programs and services offered by these contractors not only 
serve as mechanisms for mediating and implementing policy, but may also play a bro¬ 
kering role in translating research into practice. Flowever, there has been no discussion 
or evidence to date about the degree or quality of research use in their products, in 
spite of increased expectations for evidence-based decision making and practice. 
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This discussion should be part of the public dialogue, since vendors are posi¬ 
tioned to have an impact on what happens in schools. Quality matters, and attention 
needs to be paid to whether the services implemented are likely to improve teaching 
and learning. We highlight three salient issues: research production, agency demands 
for quality, and vendors’ use of research in developing services. 

First, findings suggest limited linkages among the research used in vendor pro¬ 
posals. We offered multiple interpretations of that finding and the implications for 
research production. Whether practice is ahead of research in the field of data use, 
or whether research is so broad and complex that users can draw on entirely separate 
bodies of work, our results should signal researchers to focus efforts on current prob¬ 
lems of practice in schools—here, implementing data use in schools. Building co¬ 
herent bodies of knowledge around issues highly relevant to school improvement 
efforts may help users—vendors, policy makers, administrators, and so on—to be 
clearer about which research can and should inform decisions and to make better 
informed choices. 

Second, this work is premised on being able to ascertain the role of research 
from vendor proposals, which are in turn organized by the demands put forth in 
the state RFP. Delaware requires discussion of the “evidence of effectiveness” and “re¬ 
search base for your methodology.” Laudably, this section carries the greatest weight 
in the scoring of proposals. This may or may not be the case for other state or local 
education agencies, which has two implications. First, it may make inquiry of this 
nature difficult to pursue, as identifying the population of vendors, assessing the re¬ 
search base from internal documents, and comparing across institutions may be ex¬ 
cessively time consuming and reliant on information not accessible to researchers. 
Second, and perhaps more importantly, whether or not RFPs incorporate demands 
for research use may indicate how the agency values research and may reflect expec¬ 
tations for the quality of proposals. Policy makers committed to evidence-based prac¬ 
tice should consider the ways that it can be incorporated into all programs, including 
those supported through educational contractors. Through tasks as simple as the 
design of the RFP, they can communicate strong expectations and begin to evaluate 
the quality of services provided to their schools and teachers in terms of the sup¬ 
porting evidence, though such an approach has implications in terms of the skills 
and time required to meaningfully evaluate incorporation of research in program de¬ 
sign and delivery. Educational agencies’ approach to RFP development and evalua¬ 
tion is thus worthy of inquiry on its own, and at the time of this analysis, no known 
studies of this process have been conducted. 

A second salient issue is that the services purchased to support schools ought 
to be grounded in evidence that implementation will improve teaching and learning. 
Our preliminary analyses indicate that not all vendors draw on research or other ev¬ 
idence, nor do they rely on a common body of evidence, in designing services. 
Rather, vendors’ use of research varies substantially, not only in the quantity of re¬ 
search, but in the diversity of research referenced, as suggested by the limited overlap 
in referenced sources. This is a red flag for policy makers investing in vendors to 
leverage large scale improvement, since a lack of supporting evidence suggests that 
the likelihood of desired outcomes is unknown. A further consideration is whether 
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the evidence presented is of sufficient quality to inform decisions—a dimension of 
research we did not feel was appropriate for the scope of this work, but which war¬ 
rants greater attention in the research use and educational policy communities. 

The purpose of this article has been to bring attention to issues of evidence use, 
using educational contractors by way of example. Findings here offer directions for 
further research; more importantly, they emphasize that issues of quality should be 
central to the policy dialogue around educational contracting and privatization in 
education. Through open discussion of these issues, we can improve the quality of 
both the process of acquiring services and the services themselves. 

Notes 

1. We have opted not to discuss the details of the winning proposal, as we were 
granted access to these documents by the Department of Education under an 
agreement about confidentiality. The winner of the competition is public 
knowledge, and the details of the proposal are not publicly available. 

2. Though the sample size is insufficient for detecting even large effects sizes 
among variables, relationships were tested using analysis of variance to detect 
statistical significance. Only one relationship produced p<.05 (the proportion 
of references used as evidence of effectiveness). 
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